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■ Abstract 
^ [ 

Oh| We present an algorithm for computing Fp, the pth moment of an n-dimensional frequency 

■ vector of a data stream, for 2 < p < log(n), to within 1 ± e factors, e S [n'^^P, 1] with high 
I constant probabihty. Let m be the number of stream records and M be the largest magnitude 

■ of a stream update. The algorithm uses space in bits 

Q ; where, E{p,n) = (1 - 2/p)-^{l - n.-^'-^-^/P). Here E{p,n) is 0(1) for p = 2 + and 

^ , O(logn) forp = 2+0(l/log(n). This improves upon the space required by current algorithms 

O ; [inilSlElle] byafactorofatleastrj(e-4/Pmin(log(n),e4/P-2)). The update time is 0(log{n)). 

We use a new technique for designing estimators for functions of the form ■0(E [X]), where, 
' X is a random variable and ■(/' is a smooth function, based on a low-degree Taylor polynomial 

J> , expansion of ^(E [X]) around an estimate of E [X]. 

(N ■ 

in ; 

^ ■ 1 Introduction 

'NT ■ 

■ Massive and continuous data is generated by varied sources, such as data network switches, 
. satellite imagery, sensor networks, web-click and transaction data, and require efficient, on-the 

^-H I fly analysis for early warning of critical or significant behavior. The data stream model is one 

^ ' of well-known computational models for massive data analysis. In this model an algorithm is 

typically given a relatively small amount of memory to summarize a large and (often rapidly 
arriving) dataset and is usually allowed a single pass (or at most a few passes) over the data. The 
algorithm must answer queries on the dataset for which it may only use the data summary. The 
model has been used in earlier work by Morris [H], Munro and Patterson [15j and Flajolet and 
Martin [9]. The streaming model became popular due to the seminal work of Alon, Matias and 
Szegedy [l] and a simultaneous growth in applications for monitoring massive data from various 
quarters, including networking and sensor networks. A survey is given by Muthukrishnan [32]. 

A data stream is viewed as a sequence of records of the form (i, v), where, i G {1, 2, . . . , n} = 
[n] and v £ [— M, . . . , M] and integral. The record (i, v) changes the frequency fi as fi fi + v. 
The vector / = [/i,/2, • • • , fnV is called the frequency vector of the stream. Let m be the 
number of records appearing in the stream. Given a function of the frequency vector of the 
stream, the problem is to compute the function using space that is sub-linear in n and m. The 
space used is one of the measures used for comparing randomized algorithm streaming algorithm. 
The time taken to process each stream update (i, v) is another measure of a streaming algorithm 
and is called update time. Finally, the time taken to report the query is the reporting time and 
is also a measure that has been used fT2]. 



X 
J3 



1 



Fp Algorithm 


Space O(words) 
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Word size is 0(log(nmM)) bits. E{p,n) = (1 - 2/p)"^(l - n-^^^-^/p)). 5-(p,n) = 
mine constant fi'c("-), where, 5^1 (n) = logn, gdn) = log(fi(c-i(n))/(l - 2/p). 

" Hss uses a slightly modified level mapping function, see Appendix lAl 
The dependence on e is improved to e~2-4/p ^ 

Figure 1: Space requirement of algorithms for e- approximation of Fp, p > 2. 



The pth moment of the frequency vector / is defined as Fp = In this work, 

we consider the following problem. Given p > 2 and e G (0,1], design an algorithm that 
outputs Fp such that Pr[|-Fp — Fp\ < eFp] > 0.6. We will only consider single-pass streaming 
algorithms. Alon, Matias and Szegedy in pj first defined this problem and presented a sub- 
linear space algorithm that used 0(n^~^/^'e~^ log(nmM)) bits. The same work also showed that 
deterministic approximation algorithms require 17 (n) space. On the space lower bound front, 
Alon et.al. in [1] first showed an ^}{n^~^^'P) bound. This was significantly improved in the 
elegant works of Bar-Yossef et.al. and Chakrabarti, Khot and Sun in [i [8] to fi(ni-2/Pe-2/P) 
bits. Woodruff in ^17j showed a bound of Q{e~'^) bits for all Fp. Recently, Jayram and Woodruff 

in [11] present an improved bound of r2(n^~^/^ logn) bits. The current lower bound for space is 
17(e-2 + ^i-2/pg-2/p _^ ^1-2/piog^) i3itg_ 

Indyk and Woodruff in [TO] presented the first algorithm with space 0(n^-2/P). Tabledllists 
the published algorithms for estimating Fp, p > 2, in chronological order, along with their space 
and update-time requirements, where, update time is the time required to process each arriving 
stream record. The hierarchical sampling technique introduced by Indyk and Woodruff in [10] 
is essentially used by all later algorithms. The Hss algorithm [5] improved the space of the 
IW algorithm to 0(n^~^/^e~^~^/^ log(n) log(mM)) words. Monemizadeh and Woodruff in [13] 
estimate Fp via an F2-sampler using space n^~^/^poly(e~^ log n); they also present multi-pass 
algorithms. An elegant recursive optimization of the IW method, particularly, for p = 2 + f7(l), 
was shown by Braverman and Ostrovsky in [6]. The technique reduces the space requirement of 
the Hss method by replacing the log(mM) term by g(p,n), where, 

gi{n) =\ogn, gc{n) = log{gc^i{n))/{l -2/p), and g{p,n) = min gdn) 

c constant 

Andoni, Krauthgamer and Onak in ^ present an elegant and novel simplification of the IW method 
The AKO technique fiattens the 0(log(mM)) level-wise sampling of the original IW algorithm 
by replacing each update {i,v) by {i,w\^^ • v) where, wis are drawn from the distribution 
fw{w) oc l/nP', for w E [l,ra^] and are pair-wise independent. Their analysis is based on Preci- 
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Figure 2: Comparison of E{p,n) with g{p,n). Let log''^''(n) denote the iterated logarithm of n 
taken c times, c is a constant. 

sion Sampling [2]. The space used is 0{v}~'^/^ e~'^~'^/^ \og{n)p'^ E{p,n)) words[l]. Here E{p,n) = 
(p/(p-2))(l-n-<i-2/p)) and is 0(1) for p = 2 + J^(l), 0(log(n)) for p = 2 + 0(1/ log(n)) and 
takes intermediate values in the remainder of the range of p (see Figure [2]) . A comparison of 
E(p,n) and g{p,n) is given in Figure [2j The IW, Hss and BO algorithms can be viewed as using 
a discretized version of the distribution fy = A/y^. These algorithms are "uniform" in that the 
same sketch structure, using different sizes, can be used to estimate Fp for different values of 
p>2. 

In this paper, we present an algorithm for estimating Fp, p > 2. The space used is 
0{n^^'^/Pe^'^ log{n)p'^E{p, n)/ min(log(n), e^/^"^)) words with word size 0(log(nmM)) bits. The 
update time is 0(log(n)). Since E{p,n) = 0{g{p,n)), this algorithm improves on the space us- 
age over previous algorithms by a factor of il(e~^/P min(log n, e'^''^"^)) > il(e~^/*'). For e^/^"^ = 
il(log(n)), the space requirement is 0(n^~^/*'e~^/P log(n)ii^(p, n)) words, and for e'*/*'^^ = 0(log n), 
the space requirement is 0{ii}~'^^^e~'^E(j), n)) words. Note that the algorithm is meaningful only 
for 2 < p < log(n) and e G [n~^/^, 1], otherwise, it uses 0(nlog(mM)) space. 

Braverman and Ostrovsky in [6] state that a modification of their algorithm for the regime 
logn = o(log(mM)) and constant e requires space 0(n"^~'^/Pe~^~^/P log(n log(mM))) words. 

Remark 1. For p = 2 + o(l), a domain reduction technique of Kane et.al. [12] may be applied 
to reduce space and time. Let N(n,e) = mm(n, (n^^^^'Pe^'^)^^) and £{n,e) = log{N(n,e)). 
The modified space and time expressions for the Hss, AKO,BO and this paper's algorithm are 
obtained by modifying the corresponding space/time expressions in Figure [1] by (a) replacing 
every occurrence of log n by £(n, e) , (b) replacing n of E(j), n) and g(p, n) by iV, and, (c) adding an 
O(loglogn) term for space. For example, for p = 2 + 0(l/log(n)), applying domain reduction 
to our algorithm gives a space bound of 0(e~^/P log^(l/e) + log log n) and a time bound of 
0(log(l/e)). 

Overview. The Taylor polynomial estimator is designed for estimating [X]) where X is 
a random variable and '0 is a function with certain smoothness properties. For suitably defined 
heavy items, the estimator has low bias, controlled variance, and uses a small (logarithmic) 
number of samples for many functions "ip. We apply this technique for Fp estimation by letting 
^{x) = . The algorithm uses a pair of variants of the Countsketch structures, denoted HH 
and TPEst, respectively, for identifying the heavy-hitters, and for estimating their pth frequency 
powers using the Taylor polynomial estimator. These structures have O(logn) hash tables, with 
the tables having 0{p'^B) buckets. Corresponding to each stream update of the form {i,v), the 
structures HH and TPEst are updated by scaling v io v ■ Ui, so that the effective frequency is 
9% = fiUi- The yi's are pair- wise independent and are drawn from the distribution y = yp 

The dependency on e is reduced to e-^-^/p [3]. 
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with density fy{y) oc y G [l,n^]. A set Hg of heavy items is identified as those whose 

effective frequency crosses a certain threshold T^-their estimates gi for \gi\ have error at most 
Tg/{Wp). It is then observed that for each heavy item i, (a) /j = gi/yi is on expectation with 
error at most mm{\fi\/{10p),Tg/(lQp))), and, (b) assuming sufficient independence of the hash 
functions, there are ri(log n) tables in TPEst where i does not collide with any other heavy item. 
The AMS sketches in these non-collision buckets give us r2(logn) estimates for The Taylor 
polynomial is applied to this set of estimates to give an estimate for \fi\^. Conditioning on 
an event G that holds with sufficiently high constant probability, |E['!?j] — \ fi\^\ < 0(e"'^^|/i|^), 
Var['i?i] < 0(|/j|^^~^T^/(p^ log(n))). is scaled by the inverse of the probability that the item 
qualifies as a heavy item and the sum is taken over the heavy items. The latter probability 
cannot be obtained accurately enough due to estimation errors. This is resolved by introducing 
a small rejection probability. 



2 Taylor polynomial estimator 

Let X be a random variable with K[X] = /i and Var[X] = cx^. The problem is to design an 
estimator for such that (1) \E[9] - < 0{eip{p)) and (2) Var[(9] = 0(eV^(/")), 

particularly when is large. Singh in [16] proposed an unbiased estimator for ^{fi) for an 
analytic function tp, given an estimate A of (i. Let Xj,j > be independent copies of X. Choose 
N from the geometric distribution, that is, Pr [A'^ = j] = 1/2^'^'^^ j £ Z"^ U {0} and define 

0(V^,A,{A,}) = 2^+i(V'W(A)/Ar!)(Xi-A)(A2-A)...(A^-A) [16] . 

Then E A)] = V(E [X]) [16] and for r/^ = E [{X - Xf] = (/x - A)^ + a^, 

y,r[e{i;,X,{X,})] = [2i;\X)-^l;\^,)]+2Z,>d^)^'' M ■ 
However, the variance is still too large compared to e'^ilj'^{fi). 

Taylor polynomial estimator. Let A^^'s be identically and independently distributed with 
E [Xj] = /X and Var[Aj] = a'^. Also let the first k + 1 derivatives of ^p exist in the region A]. 
Define the estimator as follows. 

A, k) = ^{^p, A, k, {X,-}i<,<fc) = ^^^^ U^Xi - ^) • 

j=0 ^' 1=1 

Let r?2 = E [{Xj - Xf] = + {^i - Xf. 

Lemma 1 Let the first k + 1 derivatives of ijj exist in the region [^,A]. Then, 



,2 



Proof By independence of A;'s, 



f^Q j! (fc + 1)! 
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for some A' G {/j,, A), implying the first statement of the lemma. 

V.M.E[(E^ri(X,-A,)V(E^(^-)' 



j=0 •' 1=1 0<j<k 



j=l ■'' l<j<j'<k ' 

l<j<k •' l<j<k 

where, in the second step, the 2k + 1 terms corresponding to j = cancel and in the third step 
we use rj — X\. | 

For p > and real and j non-negative integral, define (p = p{p — l){p — 2) . . . {p — j + 

Then, -^x^ = (j)^^"''- Corollary [2] applies the Taylor polynomial estimator to tp{x) = x^. The 
proof is given in Appendix lAl 

Corollary 2 Let tpix) = x^, p > 2, \fi - \fi\\ < a, \fi\ > 9pa, k > 4[log(l/e)] + 8 and 
denote i){x^, fi,k) by Then, if p is integral and k + 1 > p then the bias is 0. Otherwise, 
- l/*r| < |/iri2-^ei2_ FuHher, Var[i?i] < SpViP^"^^^^- 

Averaged Taylor polynomial estimator. Let {Xi}1^i be a family of independent and 
identical estimators with expectation and variance cr^ and let A be an estimate of /i. Let s > 16/c 
and r = 0(s). Choose independently r random permutations over [s], denoted vri, . . . ,7r,.. For 
each permutation tTj, choose a random permutation tt'- of the set vrj([/i;]) = {7rj(l), . . . , 7rj(A;)}. 
Let Tj denote vr^- o tTj. Order the elements of Tj ([/c]) in increasing order of the indices, that is, 
let Tj([A;]) = {aji,aj2, • • • ,ajfc} where aji < aj2 < . . . < ajk- The averaged Taylor polynomial 
estimator ?? is defined as follows: 

^^• = E^^^fl(^S.-^)' m>^,k,r,s,{Xi}U) = {l/r)j2^, . 

v=0 ' 1=1 j=l 

The averaged polynomial estimator is also abbreviated as '&{ip,\,k,r, s). 

Lemmas E A, fc, r, s)] = ¥.[d{^,\,k)]. For ^{x) = xP,X = fu/^ = \fi\Afi - \fi\\ < 
c, > 9pa, k = 144, s = 16k and r = 12s. Then, 

Var[^?(x^/i,A;,r,s)] < aV/s)|/i|'^~V . 

The proof of Lemma is given in Appendix 1X1 

3 Algorithm 

The algorithm uses HH(C, s) and TPEst(C, s) structures, where, s = 32 max(4[log n] +4, 144), 
e"' mm(log(n), e^/P""') (p — 2)[l — K P) 
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where, the distribution y is defined below. The HH(C, s) structure is a Countsketch(C, s) 
structure, that is, there are s hash tables Ti, . . . , each consisting of C buckets. The TPEst(C, s) 
structure is a Countsketch(C, s) structure except that, (a) the hash functions ^;'s used for 
the hash tables T;'s are 3-wise independent, and, (b) the Rademacher family {6(^)}ie[n] is 4- 
wise independent for each / and is independent across the Vs. Corresponding to each stream 
update {i,v), the HH and TPEst structures are maintained as if the update was {i,v ■ yi). The 
yiS are pair- wise independent random variables that are chosen from a (discretized version of 
the) distribution 3^ whose density function is given by fy{x) = x~^p~^^\ x G [1,^], where, 
A = Ap = p(l — R~P)~^ is the normalization constant. We will show that discretizing the 
distribution with precision O (log (nmM)) bits is sufficient. The effective frequency of an item is 
given by gi = fiyi. HH is used to obtain an approximate heavy-hitter set {Hg, {gi,sgn.{gi)}i^Hg) 
with threshold Tg and error A^, where, Hg C [n], gi and sgn(gij) are estimates for \gi\ and sgn{gi) 
respectively, such that with probability 63/64, all the following conditions hold: (1) if \gi\ > Tg 
then i G Hg, (2) if l^^l < Tg - Ag, then, i Hg, (3) \gi - \fi\\ < Ag, and (4) sga{gi) = sgn{gi). 
The parameters are: 

T, ^ {^^^) "^ A, ^ T,/im and [,^] ^ 4<1^^ (1) 

For i £ Hg, define the estimate fi of as gi/yi- Let = [2 log(2rg//j)] , where, Tg G 
[Tg, 65Tg/64]. Let H = {ieHg:yi> 2^(M/^}. 

Define NC(ifg) to hold if for every i G Hg, there is a set of at least s/2 distinct table indices 
Q{i) C [s] such that i does not collide with any other item of Hg in the buckets to which i maps 
for the table indices in Q{i). That is, 

NC{Hg) = G Hg,3Q{i) C [s], > s/2[yj G Hg \ {i},yt G ht{i) + . 

The estimator is as follows. If NC{Hg) does not hold then G = (i.e., it fails). Otherwise, 
let k = max(4[logn] + 4, 144) and r = 12s. Let A G (1 ± m)^2- 

Uii = Ti[hi{i)] ■ ■ sgE{gi)/yi, I G Q{i),Ai = {i^ii}ieQ(i), 
^i = ^{xP,fi,k,r,s/2,Ai), 

e = y\ \ieH and fi> (e2/fF2/4n)V2\ . 

^^Pr3;[y >2^(/0/2] ' J^ K 2/ J f 

Analysis 

We note that the analysis is meaningful for e in the range [n^^/^,1]. For e = o{n~^/P), the 
algorithm uses space 0(nlog(mM)), making it possible for a trivial algorithm to store the entire 
stream and return Fp exactly. 

Let G2 = Eie[n]9h Since, E[G2] = F2'Ey[yf],we have, G2 < 16F2Ey[yf] with probability 
15/16. The analysis is conditioned on the conjunction of the following events, denoted by Q. 

(1) Gc^iB) = G2< 16F2Ey [yf] , (2) Qh = Hg is a {Tg, Ag) approximate 
heavy set, and (3) NC{Hg) . 
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From properties of COUNTSKETCH, Qh holds with probabihty 63/64. Conditional on Qh and 
Gg2, \Hg\ < G2/{Tg-Agf < B{l-B/Cy^ < 5(1 + 1/(10^)2). Since the hash functions of the 
TPEst structure are 3-wise independent hence, NC{Hg) holds with probability 1 — n~^^. (see 
Appendix [Al) Hence Q holds with probability at least 1 — 1/64 — 1/16 — n"^" > 58/64. 

Let ^ denote the Rademacher random variables in the TPEst structure and let (^h and Ce 
denote the random bits used by the HH and TPEst structures respectively (so that Ce includes 
the random seed for and the hash functions of TPEst) and let C refer jointly to all these random 
bits. Then, 

lE^ h« I G, Ch] = [y;'{\gi\ + Yl 9Mj)Ui)^{9i))] = ^ = 1/^1 • 

jjLi,hu{i)=hu{j) 

Lemma U] shows that as a consequence of NC(-ffg), for any i,j from Hg and distinct, the 
expectation of the product of any subset of vu^s for I G Q{i) and any subset of i^ji'^s for /' G Qj 
is the product of their expectations. 

Lemma 4 E^[t?ji?j | G,Ch] = ^^["^i \ G,CH]^^[''^j I G,Ch], where, Ch is any choice of random 
bits such that i,j G H. 

Proof We first show that if i,j G Hg, i 7^ j, Ai C Q{i) and Aj C Q{j)-, then, 

IEc[n„eA.XeA, ^^u^3u I GM = 

Viu and I'ju' are independent ii u ^ u' since the inference is made from different tables. Also 
they are independent of Ch- Hence, 

Ee[n n I ^'^^^] = i/^p'^^^'i/ip^-^^^' n ^k^^,. i g]. 

li u G AiC] Aj, then, due to NC(-ffg) and since i,j €z H C Hg, hu{i) 7^ hu{j). From 4-wise 
independence of the family, it follows that [viri^jr \ G, Ch] = The above expectation 

becomes |/.r'\'^^l|/,r^\^'l(|/.||/,|)l^''^^^l = l/.^'l I/, 

Arguing in an identical manner, and noting that fi,fj are dependent only on Ch and are 
independent of ^, we have 

K€[n(^»-/0 n i^ju-f,)\G,CH] = \m-h\^^'^\\M-h('''^ (2) 

ueAi u'eAj 

Let i,j G Hg and i / j. Denote ip^^\X)/v\ by a^(A). Then, for random permutations Ti,Tj 
chosen by the averaged polynomial estimator, 

M^i^i I s,Ch] 

0<v,v'<k I I' 

= Y ay{fi)av'{fj)\\fi\ - fi\"\\fi\ - fil" 

0<v,v'<k 

= ¥.^[^i\GXH]'^i[^3\G,CH] (3) 
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The variables 1,1' run over the least v indices of Tj([A;]) and the least v' indices of Tj([/i;]), re- 
spectively. The second step follows from The values of fi,fj are dependent only on (h, 
and therefore is the same in the corresponding expressions for | Q, Ch] and [-dj \ Q, (h] ■ 
Hence, by 

l<t,t'<r 

= E^[^i\g,CH]M^j\g,CH] ■ I 

Lemma O shows that for i £ H, fi is an accurate estimate for 
Lemma 5 Given Q and i G H, \fi — < (l/(10p)|/i|. 

Proof Since i G H, \ fi\yi = \gi\ >Tg-Ag or, yi > max(l, (Tg - Ag)/\fi\). Also, 
Ag > \gi - g^\ = \fi - imvi > \fi - \m{Tg - Ag)/\f,\ 
or, \h - I/ill < f^^Jf^\ ^ (1 - iB/Cy/')-\B/Cy/'\f,\ < . 

since, Ag/Tg < {B/CY''' = l/(llp). | 

Lemma 6 Given C,h such that i £ H, Var/^^ [vn \ Q, Qh] ^ min(A^, jjo^") ■ 

Proof Since i £ H, yi > max(l, {Tg - Ag)/\fi\). Since TPEst(C) is a Countsketch structure 
and since G holds, G2 < TgB = A^C, and sgn{gi) = sgn(g(j). So 

Var^^ [m I g, Ch] = y^'Var^^ • m ■ ^idi) I G, Ch] 

f2 s / f2A2 X / f2 

Ji ^'-^2 /.2 Ji^g \ ^ ^;„f a2 H 



< minf 1, -— ) ^ < minf A2, J' < minf A^, 



\ ' {Tg-Ag)^J C - V a^{Tg-AgyJ- V (10^)2, 

Let Xi denote the indicator variable that is 1 if i G -ff and otherwise. By Lemma [U 
conditional on ^, if i G Hg and yi > 2'(-^»)/2, then, i £ H. Equivalently, Xi = u)i{yi) = 1^^2i(/i)/2 
and Ip is 1 if P is true and is otherwise. 

Lemma 7 E^^Jjjj/Pry [y > 2^0r)/^]^ I ^] = 1- 

Proof We note that fi and yi depend only on ^ \ ^ and drop the suffix from the expectation. 
Conditional on Q, the predicate yi > '^^^^^1'^ is equivalent to i € H. Actually, the condition for 
i G -ff is (1) z G Hg and (2) yi > 2^^^*^^'^. However, conditional on Q, the second condition implies 
the first and hence the equivalence. Further, if i £ H, then fi G it c), where, c = by 

Lemma [5l 

Simple Case: l{s) is constant in the interval s G [|/i|(l — c), |/j|(l + c)]. Then, 



E 



^Pry[y > 2'(A)/2] 'J y> 2'(I.AI)/2 



^ :Pr[x, = l|g] = l . 
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General Case: l{s) is not constant in the interval — c), + c)]. Let 0(s, y) denote the 

probabiUty density function that i/i = y and fi = s, conditional on Q. Such a function (j) exists 
since there is a non-trivial probability that i G H conditional on Q for each i G [n]. So 



We have 



E 



Pry [y > 2^/0/2] 



(l){y,s)ui{y) 
y Pry [y' > 2'(«)/2] 



dsdy 



(4) 



l{s) = [log ^] G [log jj^^] G [log - log(l ± c)] G /2i] = {Zh, /i^+i} 

since (a) c < 1/20 and therefore, l2i = lu + l and (b) l{s) is an integer. 

If sy < 2fg, then, y < 2fg/s < 2'(*)/2 ^nd so 0(y, s) = 0. Hence, the region of integration is 
restricted to i? = {{s,y) : sy > 2Tg}. Sub-divide R into (1) Ri : {s,y) G R and Z(s) = lu, and, 
(2) i?2 : (s^y) £ and l{s) = l2i- Let (l)i{s,y) denote the joint probability density of s,y given 
that (s,y) G conditional on ^. Likewise, define (/>2(s,y). Let p = Pr[(s,y) G -Ri | ^] and let 
q = 1 — p. From ([4]), we have, 



E 



X?; 



Pry[y > 2'(^/2] 



<P{s,y)ui{y)dsdy 



(pi{s,y)u}i{y)dsdy 
,y)eR^ Pry[y' >2h./^] 



+ 



(t)2{s,y)uji{y)dsdy 



p-Pr[xi = l\ l{fi) = lu]_ _^q-Pr[xi = l\ iCh) = h^] 



p + q 



Pry[?/' > 2'i«/2] Pr^ [y' > 2'2i/2] 

Letft(/0 = Pry[yi>2'{/>)/2]. 
Lemma 8 For i,j G [n] and distinct, E^\^[^^ | ^] = 1. 

Proof Suppose that /(sj) and Z(sj) are respectively constant for G |/j|(libc) and Sj G |/j|(libc) 
(i.e., the simple case of the previous proof). 



E 



_ Pr[y, >2^(l/>l)/2,y,.>2W^D/2] 
Pr [y. > 2'(IM/2)] Pr [y,> 2^(1/^1/2)] 



PiUi)pjUj) 



p^m)pM\) 



p^m)pM\) 



1 



where step 2 follows by the pair-wise independence of yi and yj. The other cases proceed 
analogously. | 

Assume Q and let tjj{x) = . Suppose i £ H holds for the given (^h- Then, 

(1) Ei:[iyu I QXh] = \MJ e Q{i) (2) \gi\ >Tg-Ag> Tg{l - l/(llp)) 
(3) \fi - m\ < |/^|/(10p), and (4) Var^l^ \ GXh] < min(A2, |/,|2/(l0p)2) 
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Hence, all the premises of Corollary [2] and Lemma [3] are satisfied, with a = Var^[z^i/ | Q,Ch]- 
Hence, by Corollary [21 E^[i?i | G^h] G ± U'^e^^). Also, Xi,pi{fi) depend only on C \ C 

and the seed for (h is a substring of the seed for \ ^. Hence, 

E[^iXi/pi I g] =E^\^[ixi/p^)E^[^, \G,C\^]] 

e (1 ± 12-^'^)E^\^ E ± 12-8ei2) (5) 

since, E[xj/pj | ^] = 1 by Lemma [71 
Let 

L = {i€H\h< (e2/fF2)'/V(2n^/')} 
^1 = G ^ I l/.l < (e2/*'F2/n)i/2|^ 

By definition, 6 = E{^^/p^{fi) \ i&L}. Since i£H,f,e |/,|(1 ± (j^^) and F2 £ {I ± ^)F2, 
hence L G Li. 

Lemma 9 For n > 2000, |E [9] - Fp \ g\ < eFp/16. 

Proof EieLil/il^ ^ n((e2/PF2)V2/nV2)p < e^^/n, since, F2 < Fp^^n^-'^/P. So, 

ie[n]\L ie[n]\Li iSLi 

< Fp(12-^e^2 + e/n) < (e/1000)Fp (6) 
where, the second step follows from Q and triangle inequality for n > 2000. | 

Let pf"" be the smallest value of p^{fi) = Pry[y > 2'(^'')/2 \ i ^ h]. Let c = j^, Hi = {i \ 
|/^| > 65r,/(64(l - c))}, a = vteM ^nd a; = Ey[y^]. 

Lemma 10 Conditional on Q, if i & Hi, then, = 1 and x-i = 1. Otherwise, 3 constant Kq 
such that for n> Ko,ie [n] \ {Hi U Li) and e > n'^/P, pf"" > (1.01) (||^) 

Proof pf"" = 1 if 2'(-''')/2 < 1 or if, ^^j^^+^^/y < 1, that is, if i £ Hi. So for i £ Hi, pi = Xi = 1 
(cond. on g). For ie[n]\ {Hi U Li), 



pf" >Pr[2/>2^^''°^^Tfel] >Pr[y>^] =(^/p)[(^) "-n-^log-^ 

2 min(log(n),e*'''^ 
g4„l-2/p log(ra) 



For . e M\ii, l/d > (6^/^i^2/n)V2). Since, T, = {WuF^/B)^' > C^^'^^^t^f' 
e > n^^/P, we have, 

\fi\J -\ n e2F2min(log(n),e4/p-2)y " ^ • 

where, K,K' are constants. So, 3 constant i^'o such that for n > Xq, {jfj) ^ ~ '^"^^ ^ 
(0.999) (^)"^. Also, A=p{l+3n-^P) and so (A/:p) < 1.001. Therefore, p™" > (1.01) (^|^ 
I 
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We have for i ^ j, 

¥.^[^i{}jXiXj/{piPj) I Q] - ¥.^[^iXi/pi I Q]E^[^jXj/ pj \ Q] 
^^[Mj I QXh] I GXh\&^[^j I GXh] 



XiXj 







PiPj L 

by Lemma m Here Ch is a choice of the value of the random string such that i,j G H. Hence, 
VarJ ^i^i/pi\G]= Yl "^^^d^iXi/PilG] . (8) 

«e[n]\L2 iG[n]\L2 

Let 

i2 = G I m < (e'/^F2/(4n))i/2} . 

Lemma 11 Lei e > n~^/P. Then, there exists a constant uq such that for n > uq, Pr[|0 — Fp| < 
eFp] > 0.8. 

Proof Conditional on G, /j G (1 ± and F2 G (1 ± ^)F2, so, L2 C L. 
Var[e I g] < Var[ ^ i?,x,/p, | = 



ie[n]\L2 



E 



[Var5[ Y ^iXi/pi\g]]+y3r^\dM Yl ^i^i/Pil^]] (9) 



ie[n]\L2 je[n]\L2 



where, the first step follows by definition and since Li C L, the second step is an identity (law 
of total variance). In the next equation, let i run over [n] \ L2. 

Varg[ Y ^^X^/P^\S]= Yl ^/^^^^iXi / Pi \ Q] = Y ^Var^[i?, |zG/7,g] 

ie[n]\L2 ie[n]\L2 ie[n]\L2 ^* 

< ^ ^(3pVs)l/il'^"'min(A2,|/,|V(10p)2), by Lemma H 

where. Step 1 follows from Hence, 

Ec\«[Var5[ Y ^i^i/Pi\S]] 

ieln]\L2 

< Y (pr)~'(3pV^)l/d'^-'min(A2,|/,|V(10p)2) 

ie[n]\L2 

< 5^|/.P^-VA^/(421ogn)+ J] \fi\P{aTgr /{4200logn) 

i&Hi ie[n]\iHiUL2) 

< F2p_2A2/(421ogn) + {aTgf Fp/ (4200 log n) 

<Fp2((e/500)2 + (e/32)Vlog(n)) . (10) 

Step 1 follows since, E^^Jxi//)^] < {I / pf'')E[xi / pi] = {1/pf''). Step 2 uses (a) Lemma [lO] 
which holds for n > ETq, (b) s > 128 log n, and, (c) pj"™ = 1 for i G f^i. Step 4 uses the 



11 



following bounds. We have, F2p-2 < Fp since p > 2 and F2 < Fp^^n^-'^^P. From the latter, 
we have, Ff^^ < Fp(ni-2/P)P/2, Since, B > lOOOwe-^n^-^/P/ min(log n, e^'/f-^)^ j^^ve, 

Fp{aTg)P = Fpil6au;F2/B)P/^ < {64:)-p/^F^ mm{ePlogP/\n),e^) < {e/SfF^ 

and 

F2p-2V'^] < F^-^/P{Wp^ujF2)/{{llpfB) < {88)-h^F^logn . 
Proceeding analogously for the second summation term in ([U]), 

Var^\J ^ E^[^iX,/pi]] 

ie[n]\Li 
ie[n]\Li 
i<=ln]\Li 

< (1.001)|/,p^'Var^\J(x,//,,)] 

ieln]\Li 
iG[n]\{HiULi) 
i 

< [e/SfF'p (11) 

Step 1 follows since are independent of ^, and since Xj is boolean. Step 2 follows from 

Corollary [2j Step 3 is inferred as follows. 

Var^\J Y (x.M)|/.r(l + 12-8ei2)] = ^ (|/,|P(1 + 12-8ei2)2var^^J^^/p^] 

ie[n]\Li ie[n]\Li 

+ 25^|/^n/jr(l + 12-^'')'[Ecuhx,/(p,,p,)] -E^\Jx,M]E^\Jx,/a]] 
= E (l/.r(l + 12-'e^')'Var^\Jx,M] 

ie[n]\Li 

< E l-001|/.P^Var^\Jx,/p,] . 

ie[n]\Li 

where, the cross term vanishes by Lemma El Step 4 in the sequence of steps leading to (llip 
uses Var^\^ [xj/pi] < E^\^[xj//9f] < /9™'°E^\^ [xj/pi] = /O™™. Step 5 uses Lemma [T0| and Step 6 
is a direct simplification and the final step uses the calculation for Fp{aTgY done earlier in this 
proof. 

Adding ^ and ([H]), Var[e] < {eFp/l.bf. Hence, for n > = max(Ko, 2000), by 
Chebychev's inequality, 

Pr[|G - E[e] I < (e/L01)Fp | g] > 1/55 . 
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By dl]), |E[e I g] - Fp\ < (e/1000)Fp for n > 2000. Hence by triangle inequality, 

I© — Fp\ ^ ^Fp with probability 1 — 1/55, n > uq . 

The above probabilities were conditioned on Q, which holds with probability 57/64. So |0— Fpl < 
eFp holds unconditionally with probability at least (54/55) (57/64) > 0.8. | 

Discretizing y. We note that p = o(log(n)), otherwise, the space required is 0,(nlog{mM)) 
and a trivial linear space structure can be kept. Since |/y(x)| G [R~^p^'^\ 1] = [n^^^'^/P, 1] and 
R = ri^/P, it suffices to have a precision of 0(log(pn)) = 0(log(n)) bits each before and after 
the binary point. The resulting word size of each entry of HH and TPEst is 0(log(nmM)) bits. 
We have basically proved the main result of the paper. 

Theorem 12 For each p > 2 and < e < 1, there exists a sketch structure that can be updated 
over data streams and on query can return Fp satisfying \Fp — Fp\ < eFp with probability 0.80. 
The space used in bits is 

( , ^ (p/(p-2))(l -n-^(i-2/P))p2^i-2/p(logn)log(nmM) 
mini nlog(mM), 



min(log(n), e^/'5' 2) 
The update time is 0((logn)). 

Proof If e < vl^Ip or p > log(n), we store the stream trivially using 0{nlog{mM)) bits and 
return Fp. Otherwise, the space requirement follows from Lemma [TT] and discretization argu- 
ment above. The update time is dominated by s = O(logn) hash function evaluations. The 
hash functions are pair- wise independent for the HH(C, s) structure, 3-wise independent for 
TPEst(C, s) structure and the Rademacher sketches require pair-wise and 4-wise independence 
for the HH and TPEst structures, respectively. Thus, the update time is O(logn). | 
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A Proofs and Notes 

A slightly modified level mapping function for Hss. The variant of Hss referred to in Figure [1] uses 
the level mapping function, £ : [n] — )• [i?], where R = 4 [log (mM)]. Choose random li,. . . 
each 0(l)-wise independent hash functions [n] — ?■ {0, 1}. Define i{i) = / iff ^i(i) = 1, . . . , = 
1 and either I = R oi = 0. This avoids the need for use of Nisan's PRC. 

Probability oflSiC{Hg). Assume full independence of hash functions. For i S Hg and I £ [s], 
let Wii = 1 Hi collides with some other item in Hg in the table indexed I. Then, q = Pr [wu = 1] = 
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1 - (1 - l/(8C))l-^sl-i < |i/g|/(8C) < 2/(100p2). Let Wi = E/=i(l - wa) be the number 
of tables where i does not colhde. Then, E[Wj] > s — 2s/(100p^). By Chernoff's bounds, 
Wi > s/2 with probabihty at least 1 — e~^^'^'^ . By union bound, the probability that Wi > s/2 
for each i G Hg is at least 1 — \Hg\e'^^^'^'^ = 1 — 2Bn^^^ > 1 — 2n^^^ . Now, assuming t-wise 
independence, denote q' = Prj \^Wii = l] . A standard inclusion-exclusion argument shows that 
\q' — q\ < 2(")(g/n)* < '^{j^:^)^ ■ This is at most (l/16)g provided, t > 3. The above argument 
can be repeated using q' instead of q. Combining, we have, NC(i?g) holds with probability at 



least 1 



n 



-10 



Proof of Corollary [2} Claim about bias. Since, fi G \ fi\±a, fi G |/i|(l ± l/{9p). Further, 
^x'^~^. This function is analytic in M if p > /c and otherwise is analytic in neighbor- 



k\ ~ \k) 

hoods that do not contain 0. Since, > 1 and fi G |/j|(l it l/(9p), the function x'p is analytic 
in the interval Using LemmalU part (1), for some A' G we have. 



<\fi\ 



p 

k + l 



[1 ± l/{9p)) 



p—k—l 



my 



where the plus sign is chosen if p > k + l and otherwise the minus sign is chosen. 
Case 1: k + l>p. Then, (^^ J = if p is integral. Otherwise, 



[12) 



p 

k + l 
1 



1 



{k + l)\ 
T{p+l) 



■ [p{p-l)...{p-[p\)\ [{[p\+l-p){[p\+2-p)---{k-p)\ 
T{k + l-p) 



(A; + l)! \\T{p-[p\)\J\\T{[p\+l-p)\ 

p+l/2 



= e(2/7r)i/2(A; + l 
< e(2/^)i/2(A; + l) 



1/2 



k + l 



P 



k + l 



.-1/2 



k+1- 



Step 1 follows directly by rewriting \p{p — I) . . . [p — k)\, Step 2 follows by writing each of the 
product expressions in terms of the Gamma function. Step 3 uses the Gamma reflection formula 
|r(2:)r(l — z)| = 7r/|sin(7r2;)| > vr. Step 4 uses Stirling's approximation and Step 5 uses p < k + l. 
Applying to (fT2]) . 



p 

k + l 



[I - l/(9p)) 



(7r(A; + l))V2 
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Case 2: k + Kp. Then, | J | < (gl^. Applying to (HI 



P 

k + 1 



1 + - 

c 



d\P-k-l f d\k+l 



< 



(9p(A; + 



„(p-fc-l)/(9p) 



< e 



1/9 



12 



,12 



.9(A; + 1) 

The bias is bounded by max((e/4)i^ < 

Proof of Corollary [2lClaim about variance. We have rff = cx^ + (/j — < o"^ + o"^ 
2cr^. By Lemma [H part (2), 



V=l ^ ^ t)=l 



2 



(13) 



Taking ratios of successive terms in the summation above, we have for f>l,p>2, c>9p and 
c/d > 9p 

pV2 



101/. " {v + l){9p-l)a - 2(9p-l) 



li ^ \p - v\^/2a ^ 



< 2 



-7/2 



Substituting in (fT3]) . 



Averaged Taylor Polynomial Estimator. We restate the construction of the estimator. Let 
{Xi}f^-^ be a family of independent and identical estimators with expectation /i and variance 
and let A be an estimate of fi. Let s > 16k and r = 0(s). Choose independently r 
random permutations over [s], denoted vri, . . . , VTr-. For each permutation ttj, choose a random 
permutation ir'j of the set vrj([A;]) = {TTj{l), . . . , TTj{k)}. Let denote -KjO-Kj. Order the elements 
of Tj([A;]) in increasing order of the indices, that is, let Tj([/c]) = {aji,aj2, ■ ■ ■ ,ajk} where aji < 
aj2 < . . . < ajk- The averaged Taylor polynomial estimator -d is defined as follows: 

v=a ^' 1=1 j=l 

The averaged polynomial estimator is also abbreviated as A, fc, r, s). 
The following statement directly implies Lemma El by letting r = 16s. 

Lemma 13 (General form of Lemma [S],) E [i?(V', A, k, r)] = E [t?(^. A, A;)] . For Tp{x) = x^, \fi— 
I/ill < c^,\fi\ > 9pf7, k > 144, s > 16k, 

Var[^(xP,/„fc,r,s)] < /|/,|2p-V|/,|2p-V((3/r) + ((1 + (72)-i0)/^) 
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Proof The first statement follows by linearity of expectation. 

T 



i=l 



l<i<j<r 



Let Q'^j be the random variable that denotes the number of indices shared among the first v 

and v' positions of Ti{[k]) and Tj{[k]), that is, Q^j' = [{an, . . . ,aiv} n {flji, • • • ,aj^/}|. Let Aiy 
denote the set {an, . . . , a^} and Aj^' denote the set {aji, . . . , ajy'}. Fix an element x G Ai^. 
Then, x G Ajyi provided, there exists y G [A;] such that Trj{y) = x and Tj{y) is among the least 
v' indices among Tj([A;]). Hence, 

Pr[x G Ajy,] = (k/s) ■ iv'/k) = v'/s . 

Sampling with replacement, we obtain an upper bound Q on Q = Q'^'j . Then, Q is binomially 
distributed as Bmom.{v; v' / s) and all moments of Q are at least as large as the corresponding 
moments of Q. Let a^(A) = V'^")(A)/w!. Then, 



E 



K V V 

Wi^j] = E MX)a,,{\)E[ll{Xa,, - A) UiXa., - A) 



v,v'=0 



1=1 



k nim(v,v') 

J2 a.(A)a,.(A) E 11 - X)' 

v.v' =0 



X n E[Xt-A]Pr[g-'=g] 

t6(Ai„UA^„/)\(Ai„nA^„/) 

fc mm(v,ti') 

^ a,(A)a,,(A) ^ ri^^iti - \y+^' -^'i?r[Qjf = q] 
v,v'=0 g=0 



< J2 a^(A)a^'(A)(M-A)^+'''E^„ 

v,v' =0 
k 



< J2 a,{X)a,,iX){^i-\r+'''{l + v'/sy . 



(14) 
(15) 



v.v' =0 



Step 1 follows from the definition of -di and -dj, Step 2 separates the shared variables Xt with 
t G Aiy n Ajy/ from the exclusive variables. Step 3 is a rewriting of the previous step. Step 4 
replaces the distribution for Q by the distribution Q which uses sampling with replacement. 
Since all moments of Q are at least as large as that of Q, this step follows. The final step 
uses a property of binomial distribution, namely, if x has the distribution Binom(n;p), then 
E [b^] = (1 + (6 — l)p)"'. Since, Q^j has the distribution 'B'mom{v;v' /s), we have 



E, 



)fvv' 

■ij 







l(M-A)2^J 





1 + 



0- 

/ s 



1 + 
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We note that E [i^j^^j] is maximized when A is as far apart from as possible. Hence, ()14p is 
maximized when |/i — A| is as large as possible, which is a. Thus, 1 + ^^l^^^a^ < (l + v' /sY , 
yielding (fTSj) . 

We therefore have, 

k 

E[^i^j] -E[^i]E[^j] = J2 a,{X)a,,{\){fi - A)^+^' [(1 + v'/sY - l] 

v,v'=0 
k 



= a,{X)a,,{X){^i-\y+'''[{l + v'/sy -I] . 

since, (1 + v' / sY — 1 is if either v or v' is 0. 

Divide the range of summation 1 < v,v' < k into regions Ri : 1 < v^v' < \Vk\ and 
i?2 : Y\/k\ + 1 < v,v' < k. In the region vv' / s < k/s < 1/16. 

^ a,(A)a,, (A)(/. - A)^+^' [(1 + v'/sY - l] 

v,v'=l 

[Vk] 

= a^(A)a,/(A)(/i- Af+^'[W/s + c(W/s)2] 

v,v'=l 

= Y O'vWifJ' - XY{v/y/s + cv'^/s) , where, |c| < 1/2. (16) 

11=1 

Let ki = \_\/k\. Assuming that is analytic in A], 

X;m,(A)(/i-Ar = (^-A)^X]a.(A)(/x-Ar = (/.-A)^[ao(M)- ^ ai(A)(/. - A)*] 

v=l ^ v=Q ^ t>ki+l 

= ai(/x)(//-A)- tot(A)(^-A)* (17) 

t>fci+i 

Similarly, 

Yv'a,{X){^, - XY = Y^a,W(^L - A)'^ + (^ - xf -f^Y^,{X){^, - XY 

v=l v=l v=Q 

= a^{^J){^l - X) + a2{^l){^l - Xf - Y iV(A)(/x - A)* (18) 

t>ki+l 

Let V(x) = xP, /X = l/il, A = fi, \\fi\ - M < a, \fi\ > 9pa and k > 144. Let b = 
min([pj , fci + 1). Then, 

Y MA)(/.-A)*= Y 

t>fcl+l t>fcl+l ^ ^ 

< /r'^^- V - + + C + r + 1) I - 
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Taking ratios of successive terms of summation, we have for t > and ki = \/l44 = 12, 



(t + fci + 2)1 


{t+ki+2} 1 


(9p- l)-*~i 


{t + ki + 1) 


\( P ) 
1 Vt+fci+i/ 


(9p- 1)-* 



lp-(t + fcl + l)l ^ ^ 



Hence, 



f>fcl + l ^ ^ V 1 "T / (>Q 

<|/.r'eVV|(^^^^)|(9p)-'=Hl8/17) . 

For ki + 1 > p, < 0{{k + 1)~^/^) as shown in the proof of Corollary [5J Otherwise, 

ik^^i) < {pe/{ki + l))'''i+^ In either case, 

(18/17)6^/9 + 1)1 f , ^ ^ |(9p)"'^^ < (20/17)(18 • l2/e)-^^ < (72)-^^ . 
' V/ci + ly ' 

Similarly, it can be shown that 

E t'(flfr\\fi\ - fi)' < (20/17)(A;i + l)2(9p)-(^'^+i)|/,r-V < {72)-''\f,r'a 

Rewriting pT|) and (fTSj) and using A; > 144 



E^(^)/r'(i/.i - /.r < Pi/«rv(i + (72)-i2) 



Combining, and substituting in ()16p we have 

of (HI]) < (1 + 2(72)-")p|/;,|P-i(7)Vs . (19) 

We now consider the second region R2 : [Vk\ + 1 < V, v' < k. Then, 

k 

E a,{x)a,:{\){^i - xr+^' [(1 + v'/sy - 1] 

v,v'=ki+l 
k 



v,-u'=A:i+l \ / \ / 



2 

where, lal < 1. 



v,v'=ki+l 

E ( ) ( ) /f "^"+"'^(acT)-+-'e<'(^+^')^/(2.) (20) 
t),t)'=fci+i 
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since for each value of v' , there is a value a^, G [0, 1] such that 

(1 + v' /sY = e"^'^^^^ since, v, v < k. 

From (|20p . it follows that there exists a value of /3 G [0, 1] such that 



ii,ti'=A;i + l 

Therefore, for some < |q;|,/3 < 1 and fii,fi2 & fii: aae^^^^'^^\ we have 

k 



a,{X)a,,{X){fi - Xr+^' [(1 + v'/sY - l] 

v,v'=ki+l 



v,v'=ki+l 
k 



v=ki+l ^ ^ 

< |/^pP-V2(72)-24 . (21) 

Combining (I19p and (12ip . and since A; > 144, we have, 

E[mj] -E[i^i]E[i^j] < (l + 2(72)-")V|/i|2p-V/s + |/,|2p-2^2(72)-24 
< (1 + (72)-10)pV^I'^"V/5 . 

Therefore, for any fixed i,j with i ^ j, we have 

Var[^?] = ivar[^,] (E[^,^,] -E[^,]E[^,]) 



< |/.|'"~V2((3pVO + ((1 + (72)-10)pV^) 
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